Facilitating fluid level sensing

ABSTRACT

Apparatuses are provided to facilitate sensing fluid within a fluid system, such as a coolant-based cooling apparatus for removing heat generated by one or more electronic components. The apparatus includes a plug configured to couple to a wall of the fluid system at an opening in the wall and to form a fluid-tight seal about the opening. The plug includes a fluid a fluid-sensor-receiving space configured to receive a fluid sensor, and when the plug is coupled to the wall at the opening, to position the fluid sensor at the opening in a manner to facilitate sensing of fluid within the system. The fluid sensor is removable from the plug without requiring uncoupling of the plug from the wall. In one implementation, the fluid sensor is a proximity sensor, and the plug is fabricated of a non-conductive material, and the wall a conductive material.

BACKGROUND

The power dissipation of integrated circuit chips, and the modulescontaining the chips, continues to increase in order to achieveincreases in processor performance. This trend poses cooling challengesat the module, system, rack and data center levels.

In many large server applications, processors along with theirassociated electronics (e.g., memory, disk drives, power supplies, etc.)are packaged in removable drawer configurations stacked within anelectronics rack or frame comprising information technology (IT)equipment. In other cases, the electronics may be in fixed locationswithin the rack or frame. Conventionally, the components have beencooled by air moving in parallel airflow paths, usually front-to-back,impelled by one or more air moving devices (e.g., fans or blowers). Insome cases it has been possible to handle increased power dissipationwithin a single drawer or system by providing greater airflow, forexample, through the use of more powerful air moving devices or byincreasing the rotational speed (i.e., RPMs) of existing air movingdevices. However, this approach is becoming problematic, particularly inthe context of a computer center installation (i.e., data center).

The sensible heat load carried by the air exiting the rack(s) isstressing the capability of the room air-conditioning to effectivelyhandle the load. This is especially true for large installations with“server farms” or large banks of computer racks located close together.In such installations, liquid-cooling is an attractive technology tomanage the higher heat fluxes. The liquid absorbs the heat dissipated bythe components/modules in an efficient manner. Typically, the heat isultimately transferred from the liquid coolant to a heat sink, whetherair or other liquid. In a liquid-cooling approach, monitoring coolantlevel at one or more locations within the coolant loop may be desirable.However, this monitoring can be problematic if the liquid-cooled systemis to undergo pressurized testing.

BRIEF SUMMARY

The shortcomings of the prior art are overcome and additional advantagesare provided through the provision, in one aspect, of an apparatuscomprising a plug configured to couple to a wall of a fluid system at anopening in the wall and to form a fluid-tight seal with the wall aboutthe opening. The plug includes a fluid-sensor-receiving space configuredto at least partially receive therein a fluid sensor and, when the plugis coupled to the wall at the opening, to position the fluid sensor atthe opening in a manner to facilitate sensing of fluid within the fluidsystem, the fluid sensor being removable from the fluid-sensor-receivingspace of the plug without requiring uncoupling of the plug from thewall.

In another aspect, a system is provided which includes a coolant-basedcooling apparatus, configured to facilitate removal of heat generated byone or more electronic components, and including a coolant loopcontaining coolant of the coolant-based cooling apparatus. The systemfurther includes a fluid sense apparatus which facilitates sensing ofcoolant within the coolant loop. The fluid sense apparatus includes: aplug coupled to a wall of the coolant loop at an opening in the coolantloop wall and forming a fluid-tight seal with the wall about theopening; a fluid sensor; and wherein the plug includes afluid-sensor-receiving space configured to at least partially receivetherein the fluid sensor and to position the fluid sensor at the openingin a manner to facilitate sensing of coolant within the coolant loop.The fluid sensor is removable from the fluid-sensor receiving space ofthe plug without requiring an uncoupling of the plug from the wall ofthe coolant loop.

In a further aspect, a method is provided which includes: obtaining acoolant-based cooling apparatus to facilitate removal of heat generatedby one or more electronic components, the coolant-based coolingapparatus comprising a coolant loop containing coolant of thecoolant-based cooling apparatus; and providing a fluid sense apparatusfacilitating sensing of coolant within the coolant loop. The fluid senseapparatus includes: a plug coupled to a wall of the coolant loop at anopening in the coolant loop wall and forming a fluid-tight seal with thewall about the opening; and wherein the plug includes afluid-sensor-receiving space configured to at least partially receivetherein a fluid sensor and to position the fluid sensor at the openingin a manner to facilitate sensing of coolant within the coolant loop.The fluid sensor is removable from the fluid-sensor-receiving space ofthe plug without requiring an uncoupling of the plug from the wall ofthe coolant loop.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2 is a front elevational view of one embodiment of a liquid-cooledelectronics rack comprising multiple electronic systems being cooled viaa cooling system, in accordance with one or more aspects of the presentinvention;

FIG. 3 is a schematic of an electronic system of an electronics rack andone approach to liquid-cooling of one or more electronic componentswithin the electronic system, wherein the electronic component(s) isindirectly liquid-cooled by system coolant provided by one or moremodular cooling units disposed within the electronics rack, inaccordance with one or more aspects of the present invention;

FIG. 4 is a schematic of one embodiment of a modular cooling unit for aliquid-cooled electronics rack such as illustrated in FIG. 2, inaccordance with one or more aspects of the present invention;

FIG. 5 is a plan view of one embodiment of an electronic system layoutillustrating an air and liquid-cooling approach for cooling electroniccomponents of the electronic system, in accordance with one or moreaspects of the present invention;

FIG. 6A is a schematic of one embodiment of a partially air-cooledelectronics rack with liquid-cooling of one or more liquid-to-air heatexchangers, in accordance with one or more aspects of the presentinvention;

FIG. 6B is a partially exploded view of one embodiment of aliquid-to-air heat exchanger mounted in a rack door, which includes aheat exchanger coil and inlet and outlet plenums of a heat exchangesystem for use with an electronics rack such as depicted in FIG. 6A, inaccordance with one or more aspects of the present invention;

FIG. 7 is a schematic diagram of an alternate embodiment of a coolingsystem and coolant-cooled electronic system, which may employ modularpumping units (MPUs), in accordance with one or more aspects of thepresent invention;

FIG. 8 is a schematic diagram of a further embodiment of a coolingsystem cooling one or more electronic systems, which may employ modularpumping units (MPUs), in accordance with one or more aspects of thepresent invention;

FIG. 9 depicts an alternate embodiment of a cooling system cooling oneor more electronic systems and utilizing multiple modular pumping units(MPUs), in accordance with one or more aspects of the present invention;

FIG. 10 is a schematic diagram of one embodiment of an apparatuscomprising a modular pumping unit (MPU) and an MPU controller, inaccordance with one or more aspects of the present invention;

FIGS. 11A & 11B are a flowchart of one embodiment of a control processimplemented by a modular pumping unit (MPU) controller, in accordancewith one or more aspects of the present invention;

FIG. 12 is a flowchart of one embodiment of a control processimplemented by a system-level controller of a cooling system comprisingmultiple modular pumping units (MPUs), in accordance with one or moreaspects of the present invention;

FIG. 13 depicts one embodiment of a computer program productincorporating one or more aspects of the present invention;

FIG. 14A depicts one embodiment of a fluid sense apparatus, shown withthe fluid sensor removed from the fluid-sensor-receiving space of theplug of the fluid sense apparatus, in accordance with one or moreaspects of the present invention;

FIG. 14B depicts the fluid sense apparatus of FIG. 14A, with the fluidsensor inserted in operative position within the fluid-sensor-receivingspace of the plug, and illustrating a fluid sense window of the plugconfigured to project into a fluid system to facilitate sensing of fluidwithin the system, in accordance with one or more aspects of the presentinvention;

FIG. 14C is a schematic of one embodiment of a proximity sensor, whichmay be employed as the fluid sensor depicted in FIGS. 14A and 14B, inaccordance with one or more aspects of the present invention;

FIG. 15A depicts one embodiment of a fluid system, which may comprise acoolant reservoir of a coolant-based cooling apparatus, with a fluidsense apparatus such as depicted in FIGS. 14A and 14B partially explodedfrom the wall of the coolant reservoir, in accordance with one or moreaspects of the present invention;

FIG. 15B illustrates the fluid system of FIG. 15A, with the fluid senseapparatus operatively threaded into position to form a fluid-tight sealabout the opening in the wall of the coolant reservoir, in accordancewith one or more aspects of the present invention;

FIG. 15C is a cross-sectional elevational view of the coolant reservoirand fluid sense apparatus of FIG. 15B, taken along line 15C-15C thereof,in accordance with one or more aspects of the present invention;

FIG. 16A depicts an alternate embodiment of a fluid sense apparatus witha fluid sensor shown removed from the fluid-sensor-receiving space ofplug, in accordance with one or more aspects of the present invention;

FIG. 16B depicts the fluid sense apparatus of FIG. 16A, with the fluidsensor shown disposed within the fluid-sensor-receiving space of theplug, in accordance with one or more aspects of the present invention;

FIG. 16C is a cross-sectional elevational view of the fluid senseapparatus of FIG. 16B, taken along line 16C-16C thereof, in accordancewith one or more aspects of the present invention;

FIG. 17A depicts another embodiment of a fluid sense apparatus, with thefluid sensor shown disposed in a fluid-sensor-receiving space in anupper portion of the plug, in accordance with one or more aspects of thepresent invention; and

FIG. 17B depicts the fluid sense apparatus of FIG. 17A, and illustratingfurther one embodiment of the upper portion of the plug with the fluidsensor shown disposed in the fluid-sensor-receiving space thereof, inaccordance with one or more aspects of the present invention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, and “rack unit” are usedinterchangeably, and unless otherwise specified include any housing,frame, rack, compartment, blade server system, etc., having one or moreheat-generating components of a computer system, electronic system, orinformation technology equipment, and may be, for example, a stand alonecomputer processor having high-, mid- or low-end processing capability.In one embodiment, an electronics rack may comprise a portion of anelectronic system, a single electronic system, or multiple electronicsystems, for example, in one or more sub-housings, blades, drawers,nodes, compartments, boards, etc., having one or more heat-generatingelectronic components disposed therein or thereon. An electronic systemmay be movable or fixed, for example, relative to an electronics rack,with rack-mounted electronic drawers of a rack unit and blades of ablade center system being two examples of electronic systems (orsubsystems) of an electronics rack to be cooled. In one embodiment, anelectronic system may comprise multiple different types of electroniccomponents, and may be, in one example, a server unit.

“Electronic component” refers to any heat generating electroniccomponent of, for example, an electronic system requiring cooling. Byway of example, an electronic component may comprise one or moreintegrated circuit dies and/or other electronic devices to be cooled,including one or more processor dies, memory dies or memory supportdies. As a further example, an electronic component may comprise one ormore bare dies or one or more packaged dies disposed on a commoncarrier. Further, unless otherwise specified herein, the terms“liquid-cooled cold plate” or “liquid-cooled structure” refer to anyconventional thermally conductive, heat transfer structure having aplurality of channels or passageways formed therein for flowing ofliquid-coolant therethrough.

As used herein, an “air-to-liquid heat exchanger”, “liquid-to-air heatexchanger”, or “coolant-to-air heat exchanger” means any heat exchangemechanism characterized as described herein, across which air passes andthrough which liquid coolant can circulate; and includes, one or morediscrete heat exchangers, coupled either in series or in parallel. Anair-to-liquid heat exchanger may comprise, for example, one or morecoolant flow paths, formed of thermally conductive tubing (such ascopper or other tubing) thermally coupled to a plurality of fins acrosswhich air passes. Size, configuration and construction of theair-to-liquid heat exchanger can vary without departing from the scopeof the invention disclosed herein. A “liquid-to-liquid heat exchanger”may comprise, for example, two or more coolant flow paths, formed ofthermally conductive tubing (such as copper or other tubing) in thermalor mechanical contact with each other. Size, configuration andconstruction of the liquid-to-liquid heat exchanger can vary withoutdeparting from the scope of the invention disclosed herein. Further,“data center” refers to a computer installation containing one or moreelectronics racks to be cooled. As a specific example, a data center mayinclude one or more rows of rack-mounted computing units, such as serverunits.

One example of facility coolant and system coolant is water. However,the concepts disclosed herein are readily adapted to use with othertypes of coolant on the facility side and/or on the system side. Forexample, one or more of these coolants may comprise a brine, adielectric liquid, a fluorocarbon liquid, a liquid metal, or othercoolant, or a refrigerant, while still maintaining the advantages andunique features of the present invention.

Reference is made below to the drawings (which are not drawn to scalefor ease of understanding), wherein the same reference numbers usedthroughout different figures designate the same or similar components.

As shown in FIG. 1, in a raised floor layout of an air-cooled datacenter 100 typical in the prior art, multiple electronics racks 110 aredisposed in one or more rows. A computer installation such as depictedin FIG. 1 may house several hundred, or even several thousandmicroprocessors. In the arrangement of FIG. 1, chilled air enters thecomputer room via floor vents from a supply air plenum 145 definedbetween the raised floor 140 and a base or sub-floor 165 of the room.Cooled air is taken in through louvered covers at air inlet sides 120 ofthe electronics racks and expelled through the back (i.e., air outletsides 130) of the electronics racks. Each electronics rack 110 may haveone or more air-moving devices (e.g., fans or blowers) to provide forcedinlet-to-outlet air flow to cool the electronic components within thedrawer(s) of the rack. The supply air plenum 145 provides conditionedand cooled air to the air-inlet sides of the electronics racks viaperforated floor tiles 160 disposed in a “cold” aisle of the computerinstallation. The conditioned and cooled air is supplied to plenum 145by one or more air conditioning units 150, also disposed within datacenter 100. Room air is taken into each air conditioning unit 150 nearan upper portion thereof. This room air may comprise (in part) exhaustedair from the “hot” aisles of the computer installation defined byopposing air outlet sides 130 of the electronics racks 110.

FIG. 2 depicts one embodiment of a liquid-cooled electronics rack 200comprising a cooling apparatus. In one embodiment, liquid-cooledelectronics rack 200 comprises a plurality of electronic systems 210,which may be processor or server nodes (in one embodiment). A bulk powerassembly 220 is disposed at an upper portion of liquid-cooledelectronics rack 200, and two modular cooling units (MCUs) 230 arepositioned in a lower portion of the liquid-cooled electronics rack forproviding system coolant to the electronic systems. In the embodimentsdescribed herein, the system coolant is assumed to be water or anaqueous-based solution, by way of example only.

In addition to MCUs 230, the cooling apparatus includes a system coolantsupply manifold 231, a system coolant return manifold 232, andmanifold-to-node fluid connect hoses 233 coupling system coolant supplymanifold 231 to electronic systems 210 (for example, to cold platesdisposed within the systems) and node-to-manifold fluid connect hoses234 coupling the individual electronic subsystems 210 to system coolantreturn manifold 232. Each MCU 230 is in fluid communication with systemcoolant supply manifold 231 via a respective system coolant supply hose235, and each MCU 230 is in fluid communication with system coolantreturn manifold 232 via a respective system coolant return hose 236.

Heat load of the electronic systems is transferred from the systemcoolant to cooler facility coolant within the MCUs 230 provided viafacility coolant supply line 240 and facility coolant return line 241disposed, in the illustrated embodiment, in the space between raisedfloor 145 and base floor 165.

FIG. 3 schematically illustrates one cooling approach using the coolingapparatus of FIG. 2, wherein a liquid-cooled cold plate 300 is showncoupled to an electronic component 301 of an electronic system 210within the liquid-cooled electronics rack 200. Heat is removed fromelectronic component 301 via system coolant circulating via pump 320through liquid-cooled cold plate 300 within the system coolant loopdefined, in part, by liquid-to-liquid heat exchanger 321 of modularcooling unit 230, hoses 235, 236 and cold plate 300. The system coolantloop and modular cooling unit are designed to provide coolant of acontrolled temperature and pressure, as well as controlled chemistry andcleanliness to the electronic subsystems. Furthermore, the systemcoolant is physically separate from the less controlled facility coolantin lines 240, 241, to which heat is ultimately transferred in thisexample.

FIG. 4 depicts one detailed embodiment of a modular cooling unit 230. Asshown in FIG. 4, modular cooling unit 230 includes a facility coolantloop, wherein building chilled, facility coolant is provided (via lines240, 241) and passed through a control valve 420 driven by a motor 425.Valve 420 determines an amount of facility coolant to be passed throughheat exchanger 321, with a portion of the facility coolant possiblybeing returned directly via a bypass orifice 435. The modular coolingunit further includes a system coolant loop with a reservoir tank 440from which system coolant is pumped, either by pump 450 or pump 451,into liquid-to-liquid heat exchanger 321 for conditioning and outputthereof, as cooled system coolant to the electronics rack to be cooled.Each modular cooling unit is coupled to the system supply manifold andsystem return manifold of the liquid-cooled electronics rack via thesystem coolant supply hose 235 and system coolant return hose 236,respectively.

FIG. 5 depicts another cooling approach, illustrating one embodiment ofan electronic system 210 component layout wherein one or more air movingdevices 511 provide forced air flow 515 in normal operating mode to coolmultiple electronic components 512 within electronic system 210. Coolair is taken in through a front 531 and exhausted out a back 533 of thedrawer. The multiple components to be cooled include multiple processormodules to which liquid-cooled cold plates 520 are coupled, as well asmultiple arrays of memory modules 530 (e.g., dual in-line memory modules(DIMMs)) and multiple rows of memory support modules 532 (e.g., DIMMcontrol modules) to which air-cooled heat sinks may be coupled. In theembodiment illustrated, memory modules 530 and the memory supportmodules 532 are partially arrayed near front 531 of electronic system210, and partially arrayed near back 533 of electronic system 210. Also,in the embodiment of FIG. 5, memory modules 530 and the memory supportmodules 532 are cooled by air flow 515 across the electronics subsystem.

The illustrated cooling apparatus further includes multiplecoolant-carrying tubes connected to and in fluid communication withliquid-cooled cold plates 520. The coolant-carrying tubes comprise setsof coolant-carrying tubes, with each set including (for example) acoolant supply tube 540, a bridge tube 541 and a coolant return tube542. In this example, each set of tubes provides liquid-coolant to aseries-connected pair of cold plates 520 (coupled to a pair of processormodules). Coolant flows into a first cold plate of each pair via thecoolant supply tube 540 and from the first cold plate to a second coldplate of the pair via bridge tube or line 541, which may or may not bethermally conductive. From the second cold plate of the pair, coolant isreturned through the respective coolant return tube 542.

FIG. 6A is a schematic of another embodiment of an electronic system 600comprising a liquid-cooled electronics rack 601 with a plurality ofair-cooled electronic systems 610 disposed, in the illustratedembodiment, horizontally, so as to be stacked within the rack. By way ofexample, each electronic system 610 may be a server unit of arack-mounted plurality of server units. In addition, each electronicsystem may include multiple electronic components to be cooled, which inone embodiment, comprise multiple different types of electroniccomponents having different heights and/or shapes within the electronicsystem. As illustrated, one or more electronic systems 610 comprise anair-cooled heat sink 611 with a plurality of thermally conductive fins661 projecting from the heat sink, through which airflow through theelectronics rack passes. One or more air-moving devices 670 are providedwithin electronic system 610 to facilitate airflow from, for example, anair inlet side to an air outlet side of the liquid-cooled electronicsrack 601. As explained below, the electronics rack is liquid-cooled viathe inclusion of an air-to-liquid heat exchanger at the air outlet sideof the rack.

The cooling apparatus is shown to include one or more modular coolingunits (MCUs) 620 disposed, by way of example, in a lower portion ofelectronics rack 601. Each modular cooling unit 620 may be similar tothe modular cooling unit depicted in FIG. 4, and described above (or maycomprise multiple modular pumping units, as described below withreference to FIGS. 9-12). The modular cooling unit 620 includes, forexample, a liquid-to-liquid heat exchanger for extracting heat fromcoolant flowing through a system coolant loop 630 of the coolingapparatus and dissipating heat within a facility coolant loop 619,comprising a facility coolant supply line and a facility coolant returnline. As one example, the facility coolant supply and return linescouple modular cooling unit 620 to a data center facility cooling supplyand return (not shown). Modular cooling unit 620 further includes anappropriately-sized reservoir, pump, and optional filter, for movingliquid-coolant under pressure through system coolant loop 630. In oneembodiment, system coolant loop 630 includes a coolant supply manifold631 and a coolant return manifold 632, which facilitate flow of systemcoolant through, for example, an air-to-liquid heat exchanger 640mounted to an air outlet side (or an air inlet side) of electronics rack601. Air-to-liquid heat exchanger 640 extracts heat from airflow 648egressing from liquid-cooled electronics rack 601. By way of example,one embodiment of an air-to-liquid heat exchanger 640 is describedfurther below with reference to FIG. 6B.

FIG. 6B depicts additional details of one embodiment of an air-to-liquidheat exchanger mounted in a rack door. As shown at the left portion ofthe figure, heat exchanger 640 includes one or more tube sections 641,which in one embodiment, may have a plurality of fins projectingtherefrom. Depending upon the implementation, tube sections 641 maycomprise a single, serpentine channel, or a plurality of discrete heatexchange tube sections coupled together via inlet and outlet plenums631, 632 disposed at the edge of the rack door configured to hingedlymount to the electronics rack. As shown, the one or more heat exchangetube sections are sized to substantially cover the entire opening 645 inthe frame 644 of the door.

In the depicted embodiment, the heat exchange tube sections are fedcoolant by coolant inlet plenum 631 and exhaust coolant via coolantoutlet plenum 632. Flexible hoses (not shown) may be employed forconnecting to hard plumbing disposed near the electronics rack. Thesehoses would be brought into air-to-liquid heat exchanger 640 adjacent tothe hinge axis of the door.

FIG. 6B also illustrates one embodiment of an optional perforated planarsurface 646 is illustrated. First and second such perforated planarsurfaces 646 could be provided for covering first and second main sidesof the heat exchanger. In one embodiment, the perforated planar surfacescomprise metal plates having appropriate air flow openings to allowinlet-to-outlet airflow through the electronics rack to readily passthrough the heat exchanger. One embodiment of airflow openings in theperforated planar surfaces is depicted in FIG. 6B. In this embodiment,the perforated planar surface has a plurality of openings disposedthroughout the plate. As one example, these openings may comprisehexagon-shaped openings which maximize air flow through the perforatedsurfaces, while still providing the desired isolation of the heatexchanger.

Each heat exchange tube section may comprise at least one of acontinuous tube or multiple tubes connected together to form onecontinuous serpentine cooling channel. In the embodiment shown, eachheat exchange tube section is a continuous tube having a first diameter,and each plenum 631, 632, is a tube having a second diameter, whereinthe second diameter is greater than the first diameter. The first andsecond diameters are chosen to ensure adequate supply of coolant flowthrough the multiple tube sections. In one embodiment, each heatexchange tube section may align to a respective electronics subsystem ofthe electronics rack.

Although not shown in FIG. 6B, the heat exchange tube sections furtherinclude a plurality of fins extending from tube(s) 641 to facilitateheat transfer, for example, from air exhausted out the back of theelectronics rack to coolant flowing through the serpentine coolingchannels of the individual heat exchange tube sections. In oneembodiment, the plurality of fins comprise aluminum fins extending fromthe individual tubes, which could be constructed of copper tubing.Further, in one implementation, the fins are brazed to the tubing.

FIG. 7 illustrates another embodiment of a coolant-cooled electronicsrack and cooling system therefore, in accordance with one or moreaspects of the present invention. In this embodiment, the electronicsrack 700 has a side car structure 710 associated therewith or attachedthereto, which includes an air-to-coolant heat exchanger 715 throughwhich air circulates from an air outlet side of electronics rack 700towards an air inlet side of electronics rack 700. In this example, thecooling system comprises an economizer-based, warm-liquid coolant loop720, which comprises multiple coolant tubes (or lines) connecting, inthe example depicted, air-to-coolant heat exchanger 715 in series fluidcommunication with a coolant supply manifold 730 associated withelectronics rack 700, and connecting in series fluid communication, acoolant return manifold 731 associated with electronics rack 700, acooling unit 740 of the cooling system, and air-to-coolant heatexchanger 715.

As illustrated, coolant flowing through warm-liquid coolant loop 720,after circulating through air-to-coolant heat exchanger 715, flows viacoolant supply plenum 730 to one or more electronic systems ofelectronics rack 700, and in particular, one or more cold plates and/orcold rails 735 associated with the electronic systems, before returningvia coolant return manifold 731 to warm-liquid coolant loop 720, andsubsequently to a cooling unit 740 disposed (for example) outdoors fromthe data center. In the embodiment illustrated, cooling unit 740includes a filter 741 for filtering the circulating coolant, a condenser(or air-to-coolant heat exchanger) 742 for removing heat from thecoolant, and a pump 743 for returning the coolant through warm-liquidcoolant loop 720 to air-to-coolant heat exchanger 715, and subsequentlyto the coolant-cooled electronics rack 700. By way of example, hose barbfittings 750 and quick disconnect couplings 755 may be employed tofacilitate assembly or disassembly of warm-liquid coolant loop 720.

In one example of the warm coolant-cooling approach of FIG. 7, ambienttemperature might be 30° C., and coolant temperature 35° C. leaving theair-to-coolant heat exchanger 742 of the cooling unit. The cooledelectronic system depicted thus facilitates a chiller-less data center.Advantageously, such a coolant-cooling solution provides highly energyefficient cooling of the electronic system(s) of the electronics rack,using coolant (e.g., water), that is cooled via circulation through theair-to-coolant heat exchanger located outdoors (i.e., a dry cooler) withexternal ambient air being pumped through the dry cooler. Note that thiswarm coolant-cooling approach of FIG. 7 is presented by way of exampleonly. In alternate approaches, cold coolant-cooling could be substitutedfor the cooling unit 740 depicted in FIG. 7. Such cold coolant-coolingmight employ building chilled facility coolant to cool the coolantflowing through the coolant-cooled electronics rack, and associatedair-to-coolant heat exchanger (if present), in a manner such asdescribed above.

FIG. 8 depicts another alternate embodiment of a cooled electronicsystem which comprises an electronics rack 800 with multiple electronicsystems (or subsystems) 801, such as the coolant-cooled electronicsystems described above. An air-to-liquid heat exchanger 850 providescooled coolant via a coolant loop 851 to the electronic systems 801within electronics rack 800. A controller 860 provides energy efficientcooling control of the cooling system and electronic system and, in oneembodiment, couples to a pump 852 of air-to-liquid heat exchange unit850 to control a flow rate of coolant through coolant loop 851, as wellas to an air-moving device, such as a fan 853 associated with theair-to-liquid heat exchange unit 850. In addition to sensing pump andfan power or speed (RPMs), controller 860 is coupled to sense a targetedtemperature (T_(target)) at, for example, the coolant inputs to theindividual electronic systems 801, as well as electronic system powerbeing consumed (IT power), and the ambient airflow temperature(T_(ambient)).

FIG. 8 depicts an example of a cooled electronic system which comprisesa controller (or control system), which may implement reduced powerconsumption cooling control, in accordance with aspects of the presentinvention. Note that as used herein, a controller or control system maycomprise, by way of example, a computer or a programmable logiccontroller. The control system may include, for instance, a processor(e.g., a central processing unit), a memory (e.g., main memory), andmultiple input/output (I/O) connections, interfaces, devices, etc.,coupled together via one or more buses and/or other connections. In oneapplication, the controller or control system couples to a plurality ofsensors, such as temperature, pressure, or position sensors, as well as(optionally) to one or more actuators for controlling, for instance,coolant pump speed, fan speed, or position of one or more recirculationvalves. Note that the input/output sense and control arrangements may beintegrated within the controller or control system, or they may beexternal I/O modules or devices coupled to the controller whichfacilitate the desired sensing and actuation functions.

Typically, the heat exchanger or heat exchange assemblies employed bycooling systems such as described above in connection with FIGS. 2-8comprise conventional, non-modular, plumbing systems, which canintroduce potential leak sites, especially at locations wherefield-servicing requires coolant loops to be broken. Typically, when acoolant leak occurs in an IT rack or electronic system utilizingliquid-cooling to move the heat to a heat sink, the electronic systemneeds to be shut down for repair of the coolant leak. For example, theabove-described solutions to providing liquid-cooling to an IT rack aretypically made up of single, non-redundant components, which requireshutting down of the electronic system or rack to service and/or replacea failed or failing component. Disclosed hereinbelow are enhancedcooling systems which address this issue, and allow for servicing of thecooling system without shutting down the respective electronic system(s)or rack.

Generally stated, disclosed herein, in one aspect, is an apparatus whichcomprises a modular pumping unit (MPU) configured to couple to andfacilitate pumping of coolant through a cooling apparatus assisting inremoval of heat generated by one or more electronic systems. The modularpumping unit is a field-replaceable unit which couples to the coolingapparatus in parallel fluid communication with one or more other modularpumping units. In one embodiment, each modular pumping unit includes: ahousing; a coolant inlet to the housing; a coolant reservoir tankdisposed within the housing and in fluid communication with the coolantinlet; a coolant pump disposed within the housing and configured to pumpcoolant from the coolant reservoir tank; and a coolant outlet of thehousing, the coolant pump being coupled in fluid communication betweenthe coolant reservoir tank and the coolant outlet, wherein the coolantinlet and the coolant outlet facilitate coupling of the modular pumpingunit in fluid communication with the cooling apparatus. The apparatusfurther includes a controller associated with the modular pumping unit.The controller controls the coolant pump of the modular pumping unit,and (in one embodiment) automatically adjusts operation of the coolantpump based, at least in part, upon one or more sensed parameters.

For example, one or more coolant-level sensors may be associated withthe coolant reservoir tank to sense coolant level within the coolantreservoir tank, and the controller may automatically adjust operation ofthe coolant pump based upon the sensed level of coolant within thecoolant reservoir tank. Also, the modular pumping unit may include oneor more coolant temperature sensors disposed to sense temperature ofcoolant passing through the housing, wherein the MPU controllerautomatically adjusts an operational speed of the coolant pump basedupon coolant temperature sensed by the at least one coolant temperaturesensor. If used with a cooling apparatus comprising a coolant-to-airheat exchanger, the MPU may be disposed so that a portion of the airflowacross the coolant-to-air heat exchanger also passes through the MPU,allowing a temperature sensor to be incorporated into the MPU to sensetemperature of airflow across the liquid-to-air heat exchanger. Thissensed ambient air temperature may be employed to, for example,automatically adjust operation of the pump unit. Further details of sucha modular pumping unit are described below in reference to the exemplaryembodiment thereof depicted in FIGS. 9-12. Note in this regard, that theliquid-cooled electronic system of FIG. 9 is presented by way of exampleonly. In particular, the modular pumping units disclosed herein may beemployed with various different cooling apparatuses and systems, such asthose described above in connection with FIGS. 2-8, as discussed furtherbelow.

More specifically, disclosed herein, in part, is a modular pumping unitwhich comprises a densely integrated, field-replaceable unit, which inone embodiment, provides substantially all functional and sensor needsfor pumping and monitoring a liquid coolant used to cool, for example,one or more electronic components (such as one or more processormodules). The modular pumping unit is designed to couple, in parallelwith one or more other modular pumping units, to a cooling apparatuscomprising a heat exchange assembly, such as one or more of aliquid-to-liquid heat exchanger, a coolant-to-refrigerant heatexchanger, a coolant-to-air heat exchanger, etc., and may be locatedinternal to, for example, an IT rack, or remotely from the one or moreelectronics racks or electronic systems being cooled by the coolingapparatus. In the embodiments disclosed herein, the apparatus furthercomprises a modular pumping unit controller, as well as a system-level(or frame-level) controller. The full-functional MPU disclosed hereinprovides coolant of the proper chemistry, filtering, and monitoring, toa customer's cooling apparatus, which includes the separate heatexchange assembly, and offers the ability of the customer to reject heatfrom the coolant to (for instance) the data center's water system, or toambient air, or even to a refrigerant-based circuit, while cooling thesame rack's or system's temperature-sensitive components. Redundancy atvarious levels is readily achieved by connecting in parallel fluidcommunication two or more such modular pumping units to, for example,coolant supply and coolant return manifolds of the cooling apparatus.

FIG. 9 is a schematic diagram of one embodiment of a liquid-cooledelectronic system comprising, by way of example, an electronics rack 900with multiple electronic systems 901 liquid-cooled via a cooling systemor apparatus 910, which may be disposed internal to electronics rack 900or external, and even remote from the electronics rack. The coolingsystem comprises, in this embodiment, a coolant-to-air heat exchanger920, a coolant return manifold 930, and multiple pumping apparatuses940, 950, each comprising a modular pumping unit 941, 951, in accordancewith one or more aspects of the present invention. Advantageously, themodular pumping units 941, 951 are controlled to pump coolant throughcoolant-to-air heat exchanger 920 for distribution via the heatexchanger to, for example, one or more liquid-cooled cold plate (notshown) associated with the respective electronic systems 901. In thisembodiment, the heat exchanger assembly is cooled by ambient air 922,with an airflow being provided by one or more air-moving devices 921. Asexplained further below, an MPU controller 1 942 is associated withfirst MPU 941, and an MPU controller 2 952 is associated with second MPU951. The MPU controllers themselves facilitate cooling system controlvia a system-level controller 960.

In operation, heat generated within the electronic systems 901 isextracted by coolant flowing through (for example) respective coldplates, and is returned via the coolant return manifold 930 and theactive modular pumping unit(s), for example, MPU #1 941 (in one example)to the coolant-to-air heat exchanger 920 for rejection of the heat fromthe coolant to the ambient air passing across the heat exchanger. Inthis example, only one modular pumping unit need be active at a time,and the MPU redundancy allows for, for example, servicing or replacementof an inactive modular pumping unit from the cooling system, withoutrequiring shut-off of the electronic systems or electronics rack beingcooled. By way of specific example, quick connect couplings may beemployed, along with appropriately sized and configured hoses to couple,for example, the heat exchanger, cold plates, return manifold, andpumping units. Redundant air-moving devices 921, with appropriate drivecards, may be mounted to direct ambient airflow across thecoolant-to-air heat exchanger. These drive cards may be controlled bysystem-level controller 960, in one embodiment. By way of example,multiple air-moving devices may be running at the same time.

The MPU controllers associated with the respective MPUs may be disposedon or within the respective MPU or, for example, associated with theMPU. In one embodiment, the MPU controllers can turn on/off therespective coolant pumps, as well as adjust speed of the coolant pump.The state of the MPU is relayed by the MPU controller 942, 952 to thesystem-level controller 960. The system-level controller 960 providessystem level control for, at least in part, the cooling system. Thesystem-level controller may be disposed, for example, within theelectronics rack 900, or remotely from the electronics rack, forexample, at a central data center location. As described below, thesystem-level controller determines, in one embodiment, when switchoverof MPUs is to be made and, for example, determines when an MPU has adefect requiring switchover to a redundant MPU for replacement of thedefective MPU.

As noted, although depicted in FIG. 9 with respect to a coolant-to-airheat exchanger, the field-replaceable, modular pumping units disclosedherein may provide pumped coolant (such as water) for circulationthrough various types of heat exchange assemblies, including acoolant-to-air heat exchanger, a liquid-to-liquid heat exchanger, arack-mounted door heat exchanger, a coolant-to-refrigerant heatexchanger, etc. Further, the heat exchange assembly may comprise morethan one heat exchanger, including more than one type of heat exchanger,depending upon the implementation. The heat exchange assembly, or moregenerally heat rejection device, could be within the liquid-cooledelectronics rack, or positioned remotely from the rack.

The modular pumping unit(s) comprises a recirculation coolant loopwhich: receives exhausted coolant from the electronics rack into acoolant reservoir tank, pressurizes the coolant via a coolant pump (suchas a magnetically coupled pump), passes the pressurized coolant througha check valve, and discharges the coolant back to the electronic systemsof the electronics rack via the heat exchange assembly.

FIG. 10 is a schematic diagram of one embodiment of a modular pumpingunit, which may be employed with, for example, the cooling apparatusdescribed above in connection with FIG. 9. In the embodiment illustratedin FIG. 10, modular pumping unit 1000 comprises a housing 1010 with acoolant inlet 1011 and a coolant outlet 1013. (In one implementation,housing 1010 may comprise a fluid-tight housing.) A coolant inlet quickconnect coupling 1012 at coolant inlet 1011 and a coolant outlet quickconnect coupling 1014 at coolant outlet 1013 are provided forfacilitating coupling of the MPU to, for example, a cooling apparatussuch as described above in connection with FIG. 9.

The modular pumping unit 1000 further comprises a coolant loop 1001within the housing through which coolant received via the coolant inletis re-circulated to the coolant outlet. As illustrated, coolant loop1001 couples in fluid communication coolant inlet 1011 to a coolantreservoir tank 1015 and couples coolant reservoir tank 1015 via acoolant pump 1016 to coolant outlet 1013. A check valve 1019 is alsoprovided within the coolant loop 1001 to prevent backflow of coolantinto the modular pumping unit when the modular pumping unit is off, butcoupled in fluid communication with the cooling apparatus. In oneexample, the coolant pump 1016 comprises a centrifugal pump, and aportion of the coolant pumped from coolant reservoir tank 1015 via thecoolant pump 1016 is returned via a coolant return line 1017 through acoolant filter 1018 to the coolant reservoir tank 1015. One or morecoolant fill or drain connections 1020, 1021 may be provided at housing1010 into coolant reservoir tank 1015 to, for example, facilitatefilling or draining of coolant or air from the coolant reservoir tank,and thereby facilitate field-replaceability of the modular pumping unitin parallel fluid communication with one or more other modular pumpingunits, without requiring shutdown of the respective electronic systemsor electronics rack being cooled.

Advantageously, modular pumping unit 1000 further comprises multiplesensors, and has associated therewith an MPU controller 1030 forfacilitating automated monitoring of coolant passing through the MPU, aswell as operation of the MPU itself. In the depicted embodiment, modularpumping unit 1000 comprises, for example, a lower-level coolantreservoir sensor LV1, an upper-level coolant reservoir sensor LV2, anoutlet pressure sensor P1, a coolant flow rate sensor F1, multiplecoolant temperature sensors T1, T2 & T3, an ambient airflow temperaturesensor T4, and a coolant leak sensor LK1. In one embodiment, thesesensors are disposed within the MPU and allow the controller to control,for example, operation and/or an operational speed of coolant pump 1016,in order (for example) to provide an appropriate level of cooling to theelectronic systems or rack. The MPU controller reads the sensed valuesand responds to the sensor values, along with providing diagnosticinformation to the system-level controller (such as described above inconnection with FIG. 9). The sensors also provide information which canassist in the initial filling of the modular pumping unit, and thecooling system, and can indicate the need to, for example, top off acoolant level or to remove air pockets, as well as provide an indicationthat the coolant pump does not have sufficient coolant, requiring thecoolant pump to be shut off to prevent damage. The sensors also providediagnostic information to the system-level controller which can be usedto determine, for example, the operational state of the modular pumpingunit, and to act on that information.

FIGS. 11A & 11B depict one embodiment of a control process implemented,for example, by an MPU controller of a modular pumping unit, such asdescribed above in connection with FIGS. 9 & 10. Upon initiating MPUcontrol 1100, the MPU controller obtains (for example, every t1 seconds)current sensor readings of the associated modular pumping unit 1105.Processing determines whether the leak sensor (LK1) indicates that thereis a coolant leak 1110. If “yes”, then the controller shuts off theMPU's coolant pump, and signals the system-level controller that thereis a coolant leak 1115 (at which point the system-level controllerswitches the redundant modular pumping unit (or one of the redundantunits) on to take over the pumping load for the cooling apparatus).Assuming that the leak sensor (LK1) does not indicate a coolant leak,then the MPU controller provides the system-level controller with a noleak status indication 1120.

The control process also determines whether both level sensors in thecoolant reservoir tank indicate the presence of coolant 1125. If “no”,then processing determines whether the lower-level sensor indicates thepresence of coolant 1130, and if “no” again, determines whether theupper-level sensor indicates the presence of coolant 1135. If neithersensor indicates the presence of coolant, then the MPU controllerprovides a no coolant indication to the system-level controller, andshuts off the MPU's coolant pump 1140. Alternatively, if the upper-levelsensor indicates the presence of coolant but not the lower-level sensor,then a bad coolant level signal is provided to the system-levelcontroller, since an invalid sensor state has been identified 1145. Ifthe lower-level sensor indicates the presence of coolant but not theupper-level sensor, then a bad coolant level indication is provided tothe system-level controller, indicating that coolant needs to be addedto the coolant reservoir tank 1150. If both level sensors indicate thepresence of coolant, then a good coolant level indication is provided tothe system-level controller 1155.

Additionally, the MPU controller may provide a coolant outlet pressurereading and a coolant flow reading to the system-level controller, forexample, for diagnostic purposes 1160. The MPU controller may alsodetermine the temperature of the coolant flowing, for example, to theMPU outlet 1165 (see FIG. 11B). This may be ascertained via a singletemperature sensor, or multiple temperature sensors. In the embodimentof FIG. 10, three temperature sensors T1, T2, & T3, are employed. Avalid average temperature for these temperature sensors may be created.Any value outside a possible acceptable range would not be included inthe average, and if obtained, a bad status indication may be provided bythe MPU controller to the system-level controller. In oneimplementation, the temperature differences may be ascertained (forexample, T1-T2, T1-T3, and T2-T3). If the values are below a certainthreshold, then the average of T1, T2 and T3 may be obtained. If thevalues are outside a limit or a range, then a poor coolant temperatureis identified, and an appropriate status indication is provided to thesystem-level controller. In the embodiment of FIG. 11B, the MPUcontroller determines whether the coolant temperature is within a setrange 1170, and if “no”, forwards the bad coolant temperature value(s)to the system-level controller 1175.

Advantageously, the MPU controller may also utilize coolant temperatureto adjust the coolant pump's RPMs to, for example, maintain coolanttemperature close to a desired value 1180. After this automaticadjustment of the coolant pump, processing may wait time interval t11185 before obtaining a new set of sensor readings 1105. In one example,time interval t1 may be 1 second.

FIG. 12 depicts one embodiment of processing implemented by asystem-level controller. In this example, upon initiating system-levelmonitoring and control of the MPUs 1200, processing determines whetherthe running MPU's coolant level in the coolant reservoir tank is abovean upper operational level 1205, for example, at or above theupper-level sensor in the coolant reservoir tank of FIG. 10. If “no”,then service personnel is signaled to perform a coolant fill process forthe active MPU 1210. Processing also determines whether the MPU coolantlevel is at or above a lower acceptable threshold 1215, and if “yes”,whether the running MPU's coolant flow and pressure are above acceptablerespective thresholds 1220. If either is “no”, then a spare modularpumping unit that is coupled to in parallel fluid communication with therunning MPU is started 1225, after which the previously running MPU ispowered off and replaced 1230. Processing then waits a time interval t2before again checking the coolant level within the coolant reservoirtank 1235. Assuming that the coolant level is acceptable, and that theflow and pressure readings are acceptable, the system-level controllerascertains one or more temperatures of the electronic system beingcooled 1240, and determines whether the sensed electronic systemtemperature(s) is above an upper acceptable temperature threshold 1245.If so, then the system-level controller automatically adjustsoperational speed of the active MPU's control pump to maximum to attemptreduction in the sensed system temperature 1250. After adjustingoperational speed, or if system temperature is acceptable, processingdetermines whether it is time to switch the pumping function from theactive, running MPU, to a spare MPU coupled in parallel fluidcommunication 1255. If “no”, processing waits time t2 1235 beforerepeating the processing. If “yes”, then the system-level controllerinitiates operation of a spare MPU, runs the two MPUs in parallel for aset time interval, and then deactivates the previously running MPU 1260,thereby accomplishing the switchover of the pumping load from thepreviously running MPU to the recently-started MPU. After switchingpumping operation, processing waits time t2 1235, before again repeatingthe above-described processing.

As will be appreciated by one skilled in the art, one or more controlaspects of the present invention may be embodied as a system, method orcomputer program product. Accordingly, one or more control aspects ofthe present invention may take the form of an entirely hardwareembodiment, an entirely software embodiment (including firmware,resident software, micro-code, etc.) or an embodiment combining softwareand hardware aspects that may all generally be referred to herein as a“circuit,” “module” or “system”. Furthermore, one or more controlaspects of the present invention may take the form of a computer programproduct embodied in one or more computer readable medium(s) havingcomputer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readablestorage medium. A computer readable storage medium may be, for example,but not limited to, an electronic, magnetic, optical, electromagnetic,infrared or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific examples (a non-exhaustivelist) of the computer readable storage medium include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Referring to FIG. 13, in one example, a computer program product 1300includes, for instance, one or more non-transitory computer readablestorage media 1302 to store computer readable program code means orlogic 1304 thereon to provide and facilitate one or more aspects of thepresent invention.

Program code embodied on a computer readable medium may be transmittedusing an appropriate medium, including but not limited to, wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for one or moreaspects of the present invention may be written in any combination ofone or more programming languages, including an object orientedprogramming language, such as Java, Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language, assembler or similar programming languages. Theprogram code may execute entirely on the user's computer, partly on theuser's computer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

One or more control aspects of the present invention are describedherein with reference to flowchart illustrations and/or block diagramsof methods, apparatus (systems) and computer program products accordingto embodiments of the invention. It will be understood that each blockof the flowchart illustrations and/or block diagrams, and combinationsof blocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of one or more control aspects of the present invention. Inthis regard, each block in the flowchart or block diagrams may representa module, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

In addition to the above, one or more aspects of the present inventionmay be provided, offered, deployed, managed, serviced, etc. by a serviceprovider who offers management of customer environments. For instance,the service provider can create, maintain, support, etc. computer codeand/or a computer infrastructure that performs one or more controlaspects of the present invention for one or more customers. In return,the service provider may receive payment from the customer under asubscription and/or fee agreement, as examples. Additionally oralternatively, the service provider may receive payment from the sale ofadvertising content to one or more third parties.

In one aspect of the present invention, an application may be deployedfor performing one or more control aspects of the present invention. Asone example, the deploying of an application comprises providingcomputer infrastructure operable to perform one or more control aspectsof the present invention.

As a further aspect of the present invention, a computing infrastructuremay be deployed comprising integrating computer readable code into acomputing system, in which the code in combination with the computingsystem is capable of performing one or more aspects of the presentinvention.

As yet a further aspect of the present invention, a process forintegrating computing infrastructure comprising integrating computerreadable code into a computer system may be provided. The computersystem comprises a computer readable medium, in which the computermedium comprises one or more aspects of the present invention. The codein combination with the computer system is capable of performing one ormore control aspects of the present invention.

Although various embodiments are described above, these are onlyexamples. Further, other types of computing environments can benefitfrom one or more aspects of the present invention.

As a further example, a data processing system suitable for storingand/or executing program code is usable that includes at least oneprocessor coupled directly or indirectly to memory elements through asystem bus. The memory elements include, for instance, local memoryemployed during actual execution of the program code, bulk storage, andcache memory which provide temporary storage of at least some programcode in order to reduce the number of times code must be retrieved frombulk storage during execution.

Input/Output or I/O devices (including, but not limited to, keyboards,displays, pointing devices, DASD, tape, CDs, DVDs, thumb drives andother memory media, etc.) can be coupled to the system either directlyor through intervening I/O controllers. Network adapters may also becoupled to the system to enable the data processing system to becomecoupled to other data processing systems or remote printers or storagedevices through intervening private or public networks. Modems, cablemodems, and Ethernet cards are just a few of the available types ofnetwork adapters.

In the modular pumping unit example described above in connection withFIGS. 10-12, one or more fluid sensors are employed with a coolantreservoir of the coolant loop to sense level of coolant within thereservoir. Generally, in liquid or coolant-cooled systems, a need existsto monitor, for instance, the height of the coolant within the fluidsystem, such as within a coolant reservoir, tank, manifold, etc. Thecoolant cooled system may or may not be pressurized during normaloperation. However, Underwriters Laboratory/Canadian StandardsAssociation (UL/CSA) certification testing may require pressurization ofthe fluid system to, for example, 3×-5× normal operating pressure.During such certification testing, there can be no leaks from thesystem. Therefore, a need exists in the art for a fluid sense apparatuswhich is able to accurately, reliably sense coolant level within, forexample, a coolant reservoir or manifold, while also allowing forpressurization of the fluid system (for example, during testing thereof)without leaking or malfunctioning.

Prior approaches to sensing presence of fluid within a reservoir includemechanical floats, as well as optical sensors. Unfortunately, mechanicalfloats can become contaminated and optical sensors may signal false dryindications should bubbles collect near the sensor. Another approachwould be to employ a proximity sensor to measure, for example, waterthrough the wall of a plastic tank. Generally, however, existingapplications of proximity sensors are low pressure applications.Disclosed herein with reference to FIGS. 14A-17B are various fluid senseapparatuses which employ (in one embodiment) proximity sensors tomeasure presence of fluid within the fluid system to, for example,ascertain a fluid level (e.g., within a coolant reservoir). The fluidsense apparatuses disclosed herein function without leaking,notwithstanding system/test pressures which can be 100 PSI, or higher.

Generally stated, disclosed herein is a fluid sense apparatus whichincludes a plug configured to couple to a wall of a fluid system at anopening in the wall and to form a fluid-tight seal with the wall aboutthe opening. The plug includes a fluid-sensor-receiving space configuredto at least partially receive therein a fluid sensor and, when the plugis coupled to the wall at the opening, to position the fluid sensor atthe opening in a manner to facilitate sensing of fluid within the fluidsystem. Advantageously, the fluid sensor remains separated from thefluid and is removable from the fluid-sensor-receiving space of the plugwithout requiring uncoupling of the plug from the wall. By way ofexample, the fluid sensor may comprise a proximity sensor, and the plugmay be a threaded plug capable of withstanding the required pressures,while also positioning the fluid sensor to measure or sense fluid levelwithin the fluid system. By allowing the fluid sensor to be removablefrom the fluid-sensor-receiving space, slot, recess, etc., withoutrequiring removal of the plug from the opening in the wall, the fluidsensor may be replaced, without interrupting operation of the fluidsystem. In one example, the fluid system comprises a coolant-basedcooling apparatus configured to facilitate removal of heat generated byone or more electronic components. The cooling apparatus includes acoolant flow path or loop comprising (in one embodiment) a coolantreservoir (e.g., such as depicted in connection with FIG. 10, anddescribed above). In one implementation, the plug is fabricated of anon-conductive material, the coolant reservoir is fabricated of aconductive material (e.g., stainless steel, which is able to withstandthe necessary pressurization), and the fluid sensor is a proximitysensor. The fluid sense apparatus disclosed herein advantageously allowsa proximity sensor to be employed in association with a fluid systemthat is configured to either be pressurized, or to undergo higherpressure testing/certification without leaking. The plug is configured,in one embodiment, as a threaded plug designed to withstand the requiredpressure(s), while also positioning the fluid sensor in a manner tofacilitate measurement of fluid level within the system.

As noted, and by way of example only, the fluid sensor may comprise aproximity sensor. As one specific example, the proximity sensor may be aTS-100 TouchCell™ proximity sensor, such as commercially available fromTouchSensor Technologies, LLC of Wheaton, Ill., USA. In operation, aproximity sensor typically generates an electric field, and a change inthe electric field is induced when the sensor is brought close to adense object, such as a fluid (for instance, water or other coolant).The change in electric field is sensed, causing a digital indication.Commercially available proximity sensors may be similarly sized to asecure digital (SD) memory card (e.g., 1″×¾″). Proximity sensors areused extensively in various industries, most all of which are lowpressure applications.

Referring collectively to FIG. 14A and FIG. 14B, one embodiment of afluid sense apparatus, generally denoted a 1400, is illustrated. Fluidsense apparatus 1400 includes a plug 1410 with a fluid-sensor-receivingspace 1415. Also shown is a fluid sensor 1420 sized and configured to bereceived within fluid-sensor-receiving space 1415. In this embodiment,fluid sensor 1420 is illustrated (by way of example only) as acommercially available proximity sensor. Depending on the configurationof the fluid sensor 1420, the configuration of thefluid-sensor-receiving space 1415 within plug 1410 may vary.

In the embodiment depicted, plug 1410 comprises threads 1411 whichfacilitate threading of the plug into an opening in a wall of, forexample, a coolant flow path or loop, and more particularly, a coolantreservoir or manifold, as explained further below. Plug 1410 furtherincludes frontal recesses 1412 which facilitate tightening of the plugwithin a threaded opening in a wall using a pronged tool (not shown). Afluid sense window 1416 is provided in plug 1410. In the depictedembodiment, fluid sense window 1416 is tongue-shaped so as to extendinto the fluid system when the plug is coupled to the wall of the fluidsystem at an opening therein. As shown in FIG. 14B, the fluid sensor,which as noted, may be a proximity sensor in one example, resides atleast partially within the tongue-shaped fluid sense window 1416 so thatdepending upon the level of fluid within the fluid system, fluid (suchas water or other coolant) may surround the fluid sense window 1416 oneither side of the window. This configuration thus facilitates proximitysensing of the fluid through the window. Note that the tongue-shapedfluid sense window, and thus the fluid-sensor-receiving space whichextends (in this embodiment) into the tongue-shaped fluid sense window,may be oriented horizontally, vertically, or any angle in-between whenthe plug 1410 is operatively inserted into an opening in a wall of afluid system. In FIGS. 14A & 14B, the tongue-shaped fluid sense windowis shown, by way of example only, to extend horizontally (while in FIGS.15A-15C, the tongue-shaped fluid sense window is shown, by way offurther example, to extend vertically). Note also that, in oneimplementation, the fluid sense window 1416 is fabricated of anon-conductive material to facilitate proximity sensing by a proximitysensor disposed within the fluid-sensor-receiving space 1415 of plug1410.

FIG. 14C is a schematic of one embodiment of a fluid sensor 1420configured as a proximity sensor. In this embodiment, the proximitysensor, which may comprise a TS-100 TouchCell™ sensor such as marketedby TouchSensor Technologies, LLC, includes a first electrode 1421 andsecond electrode 1422 for establishing an electric field. In oneembodiment, electrode 1422 may comprise an electric field coil.Electrically coupled between first and second electrodes 1421, 1422 aretwo sensitivity resistors 1423, which provide input to a fluid sensecircuit 1424. The fluid sense circuit may comprise anapplication-specific integrated circuit (ASIC), such as availablethrough TouchSensor Technologies, LLC. The fluid sense circuitry 1424includes a power input 1425, receiving, for instance, a five volt input,and a signal output 1426 for providing a signal indicative of whetherfluid is present in the vicinity of the proximity sensor. In operation,the ASIC generates an electric field, senses a field disruption due tothe presence of fluid, and creates an output. Sensitivity of the sensoris determined, at least in part, via the sensitivity resistors 1423.

FIGS. 15A-15C depict one embodiment of a portion of a fluid system, suchas for instance, one or more of the above-described coolant-basedcoolant apparatuses or systems. In particular, FIGS. 15A-15C depict oneembodiment of a coolant reservoir 1500 such as might be employed in thecoolant loop of the above-described cooling apparatus of FIGS. 10-12.Note, however, that this figure is presented by way of one example only,and not by way of limitation. The fluid sense apparatuses disclosedherein may be employed with a wide variety of fluid systems, including awide variety of fluid reservoirs, tanks, manifolds, etc., to evaluatefluid level within the system, which is also referred to herein as thecoolant loop.

Referring collectively to FIGS. 15A-15C, coolant reservoir 1500 isfabricated (in one embodiment only), of a material that is able towithstand higher pressures, such as might be required during UL/CSAcertification testing of a cooling apparatus for an electronic system orelectronics rack. In one embodiment, coolant reservoir 1500 isfabricated to withstand pressures of 100 PSI, or greater. Coolantreservoir 1500 includes a coolant inlet 1501, a coolant outlet 1502, anda fill port 1503, which may be employed to provide coolant within thereservoir, and thus, within the fluid system or coolant loop whichincludes the reservoir. As one example, the coolant may comprise wateror an aqueous-based solution.

In the depicted embodiment, coolant reservoir 1500 includes a wall 1505within an opening 1510 therein sized to receive plug 1410, as oneexample. In this implementation, a threaded insert 1520 is providedabout opening 1510 to facilitate threading of plug 1410 into opening1510 in coolant reservoir 1500. In one implementation, threaded insert1520 may be fabricated of a metal, and welded to coolant reservoir 1500(which may also comprise a metal) in a fluid-tight manner. FIGS. 15B &15C depict fluid sensor 1420, such as a proximity sensor, inserted inoperative position within plug 1410, and plug 1410 threadably insertedinto the opening. As noted, plug 1410, may be fabricated of anon-conductive material. FIG. 15C depicts that, in this embodiment, thefluid sensor is oriented transverse to the opening when the plug ispositioned within the opening in the wall, and the fluid sensor extendsat least partially through the opening into the coolant reservoir 1500to, for instance, facilitate exposing the electric field generated bythe sensor to the inner chamber of the coolant reservoir.

FIGS. 16A-16C depict an alternate embodiment of a fluid sense apparatus1600, in accordance with one or more aspects of the present invention.Referring collectively to FIGS. 16A-16C, fluid sense apparatus 1600includes, in this embodiment, a plug 1610 with a fluid-sensor-receivingspace 1615 formed therein size and configured to accommodate a fluidsensor 1620, which (in one embodiment) may comprise a proximity sensorsuch as described above. Plug 1610 includes threads 1611 for threadablyengaging and sealing to an opening formed in a wall of a fluid system,such as a coolant loop, and more particularly, to a wall of a coolantreservoir, manifold, etc., of a coolant loop. In one embodiment, plug1610 is fabricated of a non-conductive material and is configured toallow for proximity sensing of fluid within a fluid system through afluid sense window 1616 defined in the plug adjacent to thefluid-sensor-receiving space 1615, as illustrated in FIG. 16C. In thisembodiment, plug 1610 is partially hollowed, with an opening 1617 whichfacilitates, for example, the flow of fluid within the fluid system inclose proximity to fluid sense window 1616, and thus, facilitates fluidsensing employing fluid sensor 1620. Note also that, in thisconfiguration, the fluid-sensor-receiving space 1615 is orientedsubstantially parallel to the opening in the wall (see FIGS. 15A-15C),and with the fluid sensor disposed within the fluid-sensor-receivingspace, positions the fluid sensor at least partially overlying theopening of the wall, and overlying the fluid sense window 1616 of theplug 1610. Note that, as with the embodiment of FIGS. 14A-15C, the fluidsensor 1620 may be removed from plug 1610 without removing the plug fromthe wall of the fluid system, and therefore advantageously allows forreplacement of the fluid sensor without interrupting functioning of thefluid system.

FIGS. 17A & 17B depict a further embodiment of a fluid sense apparatus1700, in accordance with one or more aspects of present invention. Inthis embodiment, the fluid sense apparatus comprises a two piece plug1710, which includes an upper portion 1706 and a threaded portion 1707.Upper portion 1706 is configured to mate or snap to threaded portion1707 and be held to the threaded portion via multiple retaining tabs orsnaps 1708. In one implementation, the upper and threaded portions 1706,1707 may comprises molded plastic portions of two piece plug 1710. Upperportion 1706 includes a fluid-sensor-receiving space 1715 (or slot)sized and configured to allow for the insertion or removal of a fluidsensor 1720 therein. As noted space 1715 is disposed between upperportion 1706 and threaded portion 1707 of plug 1710 when the plugportions are coupled together. In one embodiment, fluid sensor 1720 maycomprise a proximity sensor such as described above.

Threaded portion 1707 includes threads 1711 sized and configured tothreadably engage a threaded opening or a threaded insert affixed to anopening within a wall of a fluid system, such as a wall of a coolantreservoir or manifold, as described above. Threaded portion 1707 furtherincludes a cylindrical opening 1717 to a fluid sense window 1716 ofthreaded portion 1707. In operation, threaded portion 1707 may becoupled to the opening in the wall of a fluid system in a fluid-tightmanner, and upper portion 1706 may be attached or removed from thethreaded portion 1707 without removing threaded portion 1707 from thewall. The fluid-sensor-receiving space 1715 is sized and configured toposition the fluid sensor substantially parallel to fluid sense window1716 such that the fluid sensor at least partially overlies a portion ofthe opening in the wall. In one implementation, the plug 1710 isfabricated of a non-conductive material, and the wall or fluid systemcontaining the fluid to be sensed may be fabricated of a conductivematerial, while still allowing the fluid sensor 1720 to comprise aproximity sensor, and to be removable from the plug without breaking thefluid-tight seal between the plug and the wall.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise” (andany form of comprise, such as “comprises” and “comprising”), “have” (andany form of have, such as “has” and “having”), “include” (and any formof include, such as “includes” and “including”), and “contain” (and anyform contain, such as “contains” and “containing”) are open-endedlinking verbs. As a result, a method or device that “comprises”, “has”,“includes” or “contains” one or more steps or elements possesses thoseone or more steps or elements, but is not limited to possessing onlythose one or more steps or elements. Likewise, a step of a method or anelement of a device that “comprises”, “has”, “includes” or “contains”one or more features possesses those one or more features, but is notlimited to possessing only those one or more features. Furthermore, adevice or structure that is configured in a certain way is configured inat least that way, but may also be configured in ways that are notlisted.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiment was chosen and described in order to best explain theprinciples of one or more aspects of the invention and the practicalapplication, and to enable others of ordinary skill in the art tounderstand one or more aspects of the invention for various embodimentswith various modifications as are suited to the particular usecontemplated.

What is claimed is:
 1. An apparatus comprising: a proximity sensor; aplug configured to couple to a wall of a fluid system within an openingin the wall to seal the opening, without contacting an inner surface ofthe wall of the fluid system, the plug being hollow and including afluid sense window which projects through the wall into the fluid systemsubstantially transverse to the opening in the wall when the plug iscoupled to the wall within the opening; and wherein the plug comprises afluid-sensor-receiving space continuously extending within the plug fromone side of the wall, through the opening in the wall and into the fluidsense window within the fluid system on another side of the wall, thefluid-sensor-receiving opening being open at the one side of the wall,and the fluid-sensor-receiving space being configured to at leastpartially receive therein the proximity sensor and, when the plug iscoupled to the wall and seals the opening, with the proximity sensorwithin the fluid-sensor-receiving space of the fluid sense window, theproximity sensor extends within the plug from the one side of the wall,through the wall into the fluid sense window within the fluid system onthe another side of the wall to sense across the fluid sense windowpresence of fluid within the fluid system by emitting an electric fieldthrough the fluid sense window into the fluid system, and sensing adisruption in the electric field due to the presence of the fluid withinthe fluid system, the fluid sense window comprising a non-conductivematerial which allows the electric field to pass therethrough, andwherein the proximity sensor is removable from thefluid-sensor-receiving space of the fluid sense window without requiringuncoupling of the plug from the wall or unsealing of the opening in thewall.
 2. The apparatus of claim 1, wherein the plug threadably seals tothe wall within the opening therein.
 3. The apparatus of claim 2,further comprising a threaded insert affixed to the wall at the opening,wherein the plug threadably seals to the wall by threadably engaging thethreaded insert, and wherein the threaded insert comprises a conductivematerial.
 4. A system comprising: a coolant-based cooling apparatus tofacilitate removal of heat generated by one or more electroniccomponents, the coolant-based cooling apparatus comprising a coolantloop containing coolant of the coolant-based cooling apparatus; and afluid sense apparatus facilitating sensing of coolant within the coolantloop, the fluid sense apparatus comprising: a proximity sensor; a plugcoupled to a wall of the coolant loop within an opening in the wall andsealing the opening, without contacting an inner surface of the wall ofthe coolant loop, the plug being hollow and including a fluid sensewindow which projects through the wall into the coolant loopsubstantially transverse to the opening in the wall; and wherein theplug comprises a fluid-sensor-receiving space continuously extendingwithin the plug from one side of the wall, through the opening in thewall and into the fluid sense window within the coolant loop on anotherside of the wall, the fluid-sensor-receiving space being open at the oneside of the wall, and the fluid-sensor-receiving space being configuredto at least partially receive therein the proximity sensor and toposition the proximity sensor within the coolant loop to sense presenceof coolant within the coolant loop by emitting an electric field throughthe fluid sense window into the coolant loop, and sensing disruption inthe electric field due to the presence of the coolant within the coolantloop, the fluid sense window comprising a non-conductive material whichallows the electric field to pass therethrough, and wherein theproximity sensor extends within the plug from the one side of the wall,through the wall, and into the fluid sense window on the another side ofthe wall, and is removable from the fluid-sensor-receiving space of thefluid sense window without requiring uncoupling of the plug from thewall of the coolant loop or unsealing of the opening in the wall.
 5. Thesystem of claim 4, wherein the coolant loop comprises a coolantreservoir and the wall with the opening is a wall of the coolantreservoir of the coolant loop, and wherein the proximity sensor disposedwithin the fluid sense window senses for presence of coolant at thelevel of the opening in the wall of the coolant reservoir.
 6. The systemof claim 4, wherein the fluid sense window is a rectangular-shaped fluidsense window that extends into the coolant reservoir.
 7. The system ofclaim 5, wherein the plug threadably seals to the wall at the openingtherein, and wherein the coolant reservoir comprises a conductivematerial.
 8. A method comprising: obtaining a coolant-based coolingapparatus to facilitate removal of heat generated by one or moreelectronic components, the coolant-based cooling apparatus comprising acoolant loop containing coolant of the coolant-based cooling apparatus;and providing a fluid sense apparatus facilitating sensing for coolantwithin the coolant loop, the fluid sense apparatus comprising: aproximity sensor; a plug coupled to a wall of the coolant loop within anopening in the wall and sealing the opening, without contacting an innersurface of the wall of the coolant loop, the plug being hollow andincluding a fluid sense window which projects through the wall into thecoolant loop substantially transverse to the opening in the wall; andwherein the plug comprises a fluid-sensor-receiving space continuouslyextending within the plug from one side of the wall, through the openingin the wall and into the fluid sense window within the coolant loop onanother side of the wall, the fluid-sensor-receiving opening being openat the one side of the wall, and the fluid-sensor-receiving space beingconfigured to at least partially receive therein the proximity sensorand to position the proximity sensor within the coolant loop to sensepresence of coolant within the coolant loop by emitting an electricfield through the fluid sense window into the coolant loop, and sensingdisruption in the electric field due to the presence of the coolantwithin the coolant loop, the fluid sense window comprising anon-conductive material which allows the electric field to passtherethrough, and wherein the proximity sensor extends within the plugfrom the one side of the wall, through the wall, and into the fluidsense window on the another side of the wall, and is removable from thefluid-sensor-receiving space of the fluid sense window without requiringuncoupling of the plug from the wall of the coolant loop or unsealing ofthe opening in the wall.
 9. The method of claim 8, wherein the obtainingcomprises obtaining the coolant loop with a coolant reservoir, theopening being in a wall of the coolant reservoir of the coolant loop,and wherein the proximity sensor is disposed within the fluid sensewindow to sense for presence of coolant at the level of the opening inthe wall of the coolant reservoir.
 10. The apparatus of claim 1, whereinthe proximity sensor is a planar-shaped proximity sensor, theplanar-shaped proximity sensor comprising a planar electrode whichresides within the fluid-sensor-receiving space of the fluid sensewindow when the proximity sensor is operatively disposed within theplug.
 11. The apparatus of claim 10, wherein the proximity sensorfurther comprises a coil electrode, the coil electrode encircling, atleast in part, the planar electrode.
 12. The apparatus of claim 1,wherein the fluid sense window of the plug comprises a tongue-shapedfluid sense window which extends into the fluid system when the plug iscoupled to the wall and seals the opening.